On the Use of Holdout Samples for Model Selection

Researchers often hold out data from the estimation of econometric models to use for external validation. In this paper we examine possible rationales for this practice. For concreteness, two examples are considered. The first example (Todd and Wolpin (2008)) is taken from the microeconometrics literature. Suppose the goal is to evaluate the impact of a monetary subsidy to low-income households based on school attendance. A social experiment is conducted in which a randomly selected treatment sample of households is offered a school attendance subsidy at some level s̄, whereas no subsidy is provided to the households in the control sample. In order to determine the optimal subsidy level, it is necessary to extrapolate the subsidy effect to other treatment levels. This requires the development and estimation of models that embed behavioral and statistical assumptions (structural models). A holdout approach to the selection among competing structural models amounts to splitting a sample Y into two subsamples, Ye and Yho. The models are then estimated based on Ye, say the data from the control group, and ranked based on their ability to predict features of the holdout sample Yho, for instance the subsidy effect on the treatment group. Examples of this kind of external model validation in the context of randomized controlled trials are Wise (1985), Todd and Wolpin (2006), and Duflo, Hanna and Ryan (2011). The second example is taken from the macroeconometrics literature. In time series analysis, competing models are often ranked in terms of their performance